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ABSTRACT 

Using architectural design as a model, the article 
discusses the process of evaluation design. Like the architect, the 
evaluator faces a problem of finding forms that best fit a partially 
defined set of contextual demands. The context is the set of all 
factors which made demands upon the final design form. Like the 
architect, the evaluator* s design comes out of an analysis of the 
context. The evaluator* s working plan corresponds to the architect's 
sketches. The form and the product of evaluation design are not ' 
really physical. The ultimate work is to communicate, inform, 
educate, inspire or to produce a beneficial impact upon the 
appropriate people. By constantly monitoring what is happening during 
the evaluation, the evaluator can continually update the working plan 
and flexibly adjust the design to respond to what is needed. Out of 
the general nature of the design process, the following guidelines 
are extracted: (1) begin by not knowing what to do; (2) begin with 
what is happening; (3) work from the general to the specific; (4) use 
both analytic and integrative modes of thinking; (5) formulate 
general questions to guide the evaluation; (6) use evaluation methods 
flexibly; and (7) monitor, recycle, and rethink. (JAZ) 
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A more extensive and complex piece of art (e.g.. a 
symphony) cannot be created by starting at the begin- 

«!I?ri^ 9 ?^? ° n ln a llneat Way to tne « nd - 00 the 
contrary, it is necessary to continuously keep the 

totality in mind, and go fro. the whole to the parts and 
from the parts back to the whole. 

Morberg shulz (intensions in Architecture) 



THE DESIGN PROBLEM 



When you set out to evaluate an educational program, how do 
you know what to do? How do you decide what kinds of activities 
your study will include? what approaches or methods dS you 
consider using? 1 

.JSl* " ay ° f decldln * t<> do is to identify yourself with a 

method or an approach (-1 am an expert i„ experimental designs-) 
or (i do case study research"). This approach certainly 
resolves many methodological questions and works fine, as long as 
° ar0fUl J°,? tUdy ° nlY * ue »tion. that are a^ab" SThT 
evaluation history appear to point out that one-dimensional 

?' ^ ^ thw ar0 done «• <>"en of £ry 

limited value. Host programs are multi-dimensional and require a 
correspondingly multifaceted understanding. require a 

A second way of deciding what to do when you find yourself in 
the position or the evaluator is to settle on In overaU p"pose? 
or a set of goals, for the evaluation. This purpose can then 
guide the choice of activities and methods. S^tStfl! 
evaluator can adopt the purpose of measuring the extent to ISich 
the Program's objectives have been achieved? while this purpose 

slightly different goal (often adopted by state evaluation 
agencies) is to evaluate the extent to which the originally 
legislated intent of a program has been realized. 

There are other evaluation goals worth considering, as well. 
You might, as in an accreditation study, wish to certify a 
EFZLl 2? a i\ tY bY CO " pa 5 in * At to some external standards. 
lll£t?£ f° "!! t0 What ext «nt a program is fulfilling 
students' actual needs, or you might decide that it is most 
important to discover ways to improve the program, it is clear 
that there are many possible goals for an evaluation—many of 

iSJhJE /"S^H*' and 0116 ******* a different set of 
Ik t actlvlti e 8 - "here do we find guidelines for choosing 
*uoong them? 9 



A third way of deciding what to do is to consider the 
audience of the evaluation. By considering the needs of decision 
makers, program developers, staff , students, interested 
researchers , and other affected special interest groups, the 
evaluator nay choose nethod(s) which enhance the impact of the 
evaluation. 



Another source of guidance for the evaluator is the Program 
itself, since its nature is clearly going to have an effect on 
the kinds of methods the evaluator chooses* For example, an 
in-depth case study may be appropriate for the evaluation of a 
controversial graduate theological program, but not as useful in 
the study of the utility of a high school typing course. 



How does one then decide what to do? All of the above 
factors — the skills of the evaluator, the purpose of the 
evaluation, tho nature of the program, and the needs of the 
audiences— influence the approach and methods the evaluator may 
select. 



The evaluator typically faces a situation in which many 
factors influence and make demands upon the evaluation design. 
That is, neither the evaluation task nor the program being 
evaluated are well defined. This is exactly the kind of problem 
that designers face as they try to create functional forms that 
satisfy many different criteria. It is the general nature of the 
design problem that makes the architect a useful metaphor for the 
evaluator. 



In the following section, the process that an architect goes 
through in defining and solving a design problem is described in 
terms of a general model, which is shown to be analogous to the 
evaluation process. 



IBB ARCHITECT 



The architect is concerned with the process of design—the 
process of inventing physical things that display order and 
organisation in forms that respond to functional needs* 
Successful design requires that the architect operate both in a 
linear, analytical fashion and in a more intuitive, holistic 
mode. The architect Bust possess a wide range of technical 
skills and use the methods of many varied disciplines. 



In moving from the initial presentation of the problem to the 
development of the final design form, the architect goes through 
a process which can be reconstructed in a general way, as shown 
in Figure 1. 



Figure 1 



CONTEXT 




FORM 



Context 

The context is the name given to the multidimensional problem 
apace that the architect faces. The context is the set of all 
factors which make demands upon the final design form . The 
context includes physical, legal, psychological, and 
philosophical factors. All together these factors define the 
boundaries of possible solutions. For example, the context for 
the architect includes such factors as the client's desires, the 
functions the final form must serve, Mie resources available 
(including funds, material, personnel, time • . .), the nature of 
the site (including soil type, slope, exposure, access • • •), 
and any social or legal constraints (such as building codes, 
zoning laws, neighborhood tradition • • •). All of these factors 
are elements of the architect's problem context. While they 
influence the direction and shape of the architect's solution, 
they do not uniquely define it. 



Design 

In the first phase of the design process, the architect 
carries out an analysis of the context. Information is gathered 
and separated into components along different dimensions. For 
example, the architect may examine the •site dimensions" 



collecting information about the soil, the topography, the 
vegetation, the orientation to the sun and the wind, the 
variation with season, and so forth* Studying the context in an 
open-ended way, the architect discovers which dimensions are 
important, probes into the component factors of that dimension, 
and even begins to learn about how different components 
interact. (For example, in deciding where to place a house on 
the crest of an exposed hill, the architect may learn about the 
trade-off 8 between energy efficiency and the quality of the view.) 



In the second phase of the design, the architect begins to 
formulate outlines and pieces of the design. Priorities emerge 

and vague goals are stated (such as "the design should create an 
ambiance of excellence and tradition") . The philosophy, the 
style, and the basic elements of the design begin to take shape 
in the architect's mind. 



The Form 

The form is the realization of the design process; it is a 
solution which it is hoped fulfills those dimensions of the need 
most demanding of attention. The form is arrived at through a 
process of synthesis in which the ideas that arose in the design 
process through analysis are combined to form a coherent whole. 
Since the demands or factors influencing the form interact with 
each other, and since the final form must simultaneously satisfy 
as many of these demands as possible, a kind of gestalt or 
integrated vision of the final design is required. In achieving 
this integration, the architect may begin by sketching important 
conceptual relationships, by then moving to two-dimensional 
diagrams, and finally by producing three-dimensional models. 



The Fit 

The fit is the degree of congruence between the form and the 
context. It is a measure of acceptability. Out of the 
evaluation of the fit, judgments are made about how well the 
design has met the many demands of the context. Standards of 
consistency, integrity, and aesthetics are applied, poor designs 
lead to such judgments as "arbitrary, obsolete, incongruous, or 
dysfunctional." Successful design' re judged to be "efficient, 
compatible, energy-saving, and pre ional." Such judgments are 
based on the contextual factors dt ± most important in a given 
case. 



THE EVALUATOR 



very much like the architect, the evaluator faces ,1 problem 
of finding for as that best fit a partially defined set of 
contextual demands. The process the evaluator goes through can 
be viewed as very similar to that of the architect, as 
illustrated by Figure 2. 



Figure 2 



PROGRAM CONTEXT 




FORMS Of 



Program Context 

The context for the evaluator consists of a great variety of 

factors, including, for example, the following concerns: 

The needs of: 

• the client or sponsoring agency 

• the students or other affected audience 

• the teachers or program staff 

• other indirectly affected audiences 

The nature of: 

• the program's goals or intent 

• the program's activities 

• the educational or institutional setting 

• the political setting 



The constraints of: 



• 



tine (the program's and th* e valuator's) 
resources (including money and personnel) 
access 

social or legal requirements 

the evaluator's own evaluation skills 

the evaluation mandate 



All of these factors push and pull the evaluator in the attempt 
to create a balanced and effective evaluation design. 



Working evaluation Plan 

As the design is to the architect, so is the working 
evaluation plan to the evaluator. Like the architect, the 
evaluator's design comes out of an analysis of the context — that 
is, out of an insightful understanding of the total evaluation 
setting. The evaluator's working plan corresponds to the 
architect's sketches, and yet the working plan is a more dynamic 
entity. As the evaluation proceeds through the design phase and 
into its implementation, more is learned al out the context. 
Important factors that were unknown will emerge, and what was 
thought to be important may turn out not to be. Accordingly, the 
working plan is never fixed, but rather is continuously evolving 
and changing. 



Forms of Communication 

The form, the product of the evaluator's design, is not 
really physical. Even the "final report" is more a means to an 
end than an end in itself. What the evaluator ultimately wishes 
to do is to communicate— to inform, educate, inspire, arouse, or 
otherwise produce a beneficial impact upon the appropriate 
people. Thus, it is effective interaction that is the product, 
the final form, of a successful evaluation design. 



The fit is the degree to which the evaluator is able to 
understand and meet the demands of the situation. By constantly 
monitoring what is happening during the evaluation (a kind of 
dynamic self meter evaluation) , the evaluator can continually 
update the working plan and flexibly adjust the design to respond 
to what is needed. 



7 
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SUMMARY 
A Pew General Principles 



When evaluation 18 treated as a design problem, the vision of 
the evaluator's task is expanded, and the e valuator needs to 
become more creative, flexible, and artistic. There is less 
certainty, and rather than have a set of algorithms to follow, 
the best the evaluator can hope for are some general guidelines 
to go by* As discussed earlier, out of the general nature of the 
design process we can, with a little help from actual experience, 
extract the following general principles of "ideal" evaluation 
design: 



1. Begin by not knowing what to do. 



This takes an act of courage on the evaluator's part. Many 
times there are strong pressures (both internal and external) 
to restrict the evaluation to a predetermined task or 
method. Often the evaluator will need to educate and 
convince clients that, to meet their ultimate needs, the 
evaluation will have to be different from what they initially 
envisioned. Even if, ultimately, the evaluator does exactly 
follow a prescribed course, an open-minded look at the 
beginning can only increase the overall quality of the work 
that follows. 



2. Begin with what is happening. 

This is really a corollary to the first principle. Often the 
evaluator enters the scene blinded by someone else's 
concerns, the description of the program and its goals, or 
self-imposed ideas about what is and is not happening. All 
of these are legitimate concerns and questions— they should 
ultimately be addressed— but they are not necessarily good 
starting points. By beginning with what is happening, by 
studying the program as it is, the evaluator becomes grounded 
in a personal knowledge of the reality of the program. Then, 
through observation, discussion, document analysis, and 
serendipity, the evaluator can discover what in the setting 
(context) is and is not important. 



3* work from the general to the specific. 

Issues, questions, and important dimensions (as well as ways 
to proceed) may at first be only vaguely sensed. As in the 
design of a building, the final design will be more complete 
and successful if the evaluator's vision is allowed to mature 
slowly and change along the way. 



4. use both analytic and integrative aodes of thinking 



Analysis is needed to discover and separate out the important 
dinensions of the scene* while an integrated vision is 
required to create an encompassing vision of the evaluation 
in its entirety. And these two aodes of thought interact. 
When analysis yields a new factor to consider, the overall 
vision of the evaluation nay change, when the overall vision 
changes, new directions for analysis nay be indicated. 



5. Formulate general questions to guide the evaluation. 

Out of the analysis of the context, the evaluator begins to 
sense what is and what is not worth addressing. This sense 
can be made more specific by formulating a few general 
questions to guide the activities and methods of the 
evaluation. Questions such as "How can we understand what is 
happening in the program?" or "what are the barriers to this, 
program's effectiveness?" are at about the right level of 
detail. (This notion of using general questions to help 
direct the evaluator is explored in detail in Guide Mo. 14.) 



6. Use evaluation methods flexibly. 

The architect must draw upon and skillfully use methods from 
a wide range of disciplines (art, mechanical drawing, 
engineering, sociology . . .). All of the techniques of 
these disciplines are needed and used by the architect. 
Guided by what needs to be done, a skillful architect even 
takes delight in the creative use of many different skills 
and methods. The evaluator would do well to emulate the 
architect in terms of the flexible use of methods. 



7. Monitor, recycle, and rethink. 

For soae reason, •valuation plans sees to be cast in bronze 
early on. Even if the charted course makes no sense to 
anyone, the evaluation often proceeds as scheduled. However, 
if the evaluator begins with the idea of working with a 
sketch—a plan that is by design going to evolve and 
change — then the final evaluation is more likely to fit with 
what is needed. 
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